Image capturing apparatus

ABSTRACT

The image capturing apparatus includes a main image capturing system configured to capture an object image formed by a main optical system whose magnification is variable, and multiple sub image capturing systems configured to respectively capture multiple object images respectively formed by multiple sub optical systems. The multiple optical systems are arranged on both sides across a sectional plane including an optical axis of the main optical system. The apparatus further includes a processor configured to acquire, using a main image produced by the main image capturing system and multiple sub images produced by the multiple image capturing systems, object distance information in an image capturing view angle of the main image capturing system.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image capturing apparatus having amain image capturing system and a sub image capturing system.

Description of the Related Art

Image capturing apparatuses such as video cameras and digital camerasare proposed, each of which has, in addition to a main image capturingsystem for acquiring a normal captured image, a sub image capturingsystem. Japanese Patent Laid-Open No. 2012-049651 discloses an imagecapturing apparatus that has, in addition to a main image capturingsystem including a zoom lens, a sub image capturing system including afixed focal length lens having a wide view angle. This image capturingapparatus acquires a wider view angle captured image by the sub imagecapturing system than that acquirable by the main image capturing systemto facilitate following a moving object. Japanese Patent Laid-Open No.2012-049651 further discloses a method for acquiring information on anobject distance, using the captured images acquired by the main and subimage capturing systems.

Furthermore, Japanese Patent Laid-Open No. 2013-061560 discloses animage capturing apparatus having, in addition to a main image capturingsystem having a zoom lens, multiple stereo cameras as sub imagecapturing systems having mutually different focal lengths.

This image capturing apparatus acquires object distance information fromparallax information provided by the stereo cameras and performs AF(autofocus) of the main image capturing system using the object distanceinformation.

However, in the image capturing apparatuses disclosed in Japanese PatentLaid-Open Nos. 2012-049651 and 2013-061560, a viewpoint position of thesub image capturing system is different from that of the main imagecapturing system, which generates in the captured image acquired by thesub image capturing system an object area (occlusion area) not includedin the captured image acquired by the main image capturing system. It isgenerally difficult to acquire the parallax information in the occlusionarea. Therefore, the captured image acquired by the main image capturingsystem includes an area where the object distance information cannot beacquired from the parallax information acquired using the sub imagecapturing system.

SUMMARY OF THE INVENTION

The present invention provides an image capturing apparatus capable ofavoiding, over an entire image capturing view angle of a main imagecapturing system, problems such as impossibility of acquiring objectdistance information due to generation of an occlusion area in a subimage capturing system.

The present invention provides as an aspect thereof an image capturingapparatus including a main image capturing system configured to capturean object image formed by a main optical system whose magnification isvariable, and multiple sub image capturing systems configured torespectively capture multiple object images respectively formed bymultiple sub optical systems. The multiple optical systems are arrangedon both sides across a sectional plane including an optical axis of themain optical system. The apparatus further includes a processorconfigured to acquire, using a main image produced by the main imagecapturing system and multiple sub images produced by the multiple imagecapturing systems, object distance information in an image capturingview angle of the main image capturing system.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front view of an image capturing apparatus that isEmbodiment 1 of the present invention.

FIG. 2 is a block diagram illustrating a configuration of the imagecapturing apparatus of Embodiment 1.

FIGS. 3A to 3C illustrate captured images acquired by the imagecapturing apparatus of Embodiment 1.

FIG. 4 is a flowchart illustrating an image capture/image combinationprocess in Embodiment 1.

FIGS. 5A and 5B illustrate a method for extracting corresponding points.

FIGS. 6A to 6D illustrate a trimming/enlargement process in Embodiment1.

FIG. 7 is a flowchart illustrating an object distance calculationprocess in Embodiment 1.

FIGS. 8A to 8D illustrate use of telephoto-side images in Embodiment 1.

FIG. 9 is a front view of an image capturing apparatus that isEmbodiment 2 of the present invention.

FIGS. 10A to 10C illustrate captured images acquired by the imagecapturing apparatus of Embodiment 2.

FIG. 11 is a front view of an image capturing apparatus that isEmbodiment 3 of the present invention.

FIG. 12 is a front view of an image capturing apparatus that isEmbodiment 4 of the present invention.

FIG. 13 illustrates image capture from different viewpoint positions.

FIGS. 14A and 14B illustrate occlusion areas.

FIG. 15 illustrates a model of stereo image capturing method.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

First, description will be made of features common to the embodimentsbefore specific descriptions thereof. An image capturing apparatus ofeach embodiment has a main image capturing system configured to capturean object image formed by a main optical system whose magnification isvariable and multiple image capturing systems configured to capturemultiple object images respectively formed by multiple sub opticalsystems. The image capturing apparatus acquires from the main imagecapturing system a main image as a captured image and acquires from themultiple sub image capturing system multiple sub images as capturedimages. Using these main and multiple sub images enables avoiding, overan entire image capturing view angle of the main image capturing system,problems such as impossibility of acquiring object distance informationdue to generation of an occlusion area resulted from a differencebetween viewpoint positions of the main and sub optical systems, thatis, enables acquiring accurate object distance information over theentire image capturing view angle of the main image capturing system.

Description will be made of the occlusion area generated due to thedifference of the viewpoint positions with referring to FIGS. 13, 14Aand 14B. FIG. 13 illustrates objects A and B and two image capturingapparatuses C1 and C2 capturing the objects A and B. The image capturingapparatuses C1 and C2 are separate from each other by a base length. Theobject A is separate from the image capturing apparatuses C1 and C2 byan object distance La, and the object B is separate from the imagecapturing apparatuses C1 and C2 by an object distance Lb (>La). FIGS.14A and 14B illustrate captured image respectively acquired by the imagecapturing apparatuses C1 and C2. As understood from these drawings, ageometric relation depending on the base length and the objectdistances, the captured images include mutually different areas of theobjects A and B. For example, the captured image illustrated in FIG.14A, which is acquired by the image capturing apparatus C1, includes asurface A3 of the object A and a surface B3 of the object B. However,the captured image illustrated in FIG. 14B, which is acquired by theimage capturing apparatus C2, does not include the surfaces A3 and B3.On the other hand, the captured image is illustrated in FIG. 14Bincludes a surface B4 of the object B not included in the captured imageillustrated in FIG. 14A. As just described, an area included in one ofcaptured images acquired by image capture from mutually differentviewpoint positions but not included in the other captured image isreferred to as “the occlusion area”.

Next, description will be made of a method for calculating an objectdistance using parallax images acquired by image capture from mutuallydifferent viewpoint positions with referring to FIG. 15. FIG. 15illustrates a model of stereo image capturing method. Coordinates (x,y)are based on an origin that is a center between a left camera L_cameraand a right camera R_camera. A position in a horizontal direction isexpressed by an x-coordinate along an x-axis, and a position in a depthdirection is expressed by a y-coordinate along a y-axis. A coordinate ina height direction is omitted.

In this description, principal points of imaging optical systems of theleft camera L_camera and the right camera R_camera are located at(−Wc,0) and (Wc,0), respectively. A focal length of each of thesecameras L_camera and R_camera is represented by f. The left and rightcameras L_camera and R_camera capture an object A located at coordinates(0,y1) on the y-axis. Displacement amounts (in other words, parallaxamounts) of optical images (object images) of the object A from centersof image sensors of the left and right cameras L_camera and R_camera areexpressed by following expressions (1) and (2).

$\begin{matrix}{{Plc} = {{- \frac{wc}{y\; 1}} \cdot f}} & (1) \\{{Prc} = {\frac{wc}{{y\; 1}\;} \cdot f}} & (2)\end{matrix}$

Namely, image capture of an identical object from mutually differentviewpoint positions enables acquiring left and right parallax imagesincluding displacement amounts (parallax amounts) Plc and Prc expressedby above expressions (1) and (2) in a viewpoint separation direction(base length direction). An object distance y1 to the object A can becalculated using these displacement amounts Plc and Prc by followingexpression (3).

$\begin{matrix}{{y\; 1} = {\frac{2{wc}}{{Prc} - {Plc}} \cdot f}} & (3)\end{matrix}$

As described above, in order to calculate the object distance using theparallax images, it is necessary to specify object areas correspondingto each other (that is, object areas including an identical object) inthe parallax images. As a method for specifying such correspondingobject areas in two images, a corresponding point extraction method suchas a block matching method, which will be described later, can be used.

In the following description, the image capturing apparatus C1 asillustrated in FIG. 13 is defined as a main image capturing system, andthe image capturing apparatus C2 is defined as a sub image capturingsystem. As described above, the captured image (FIG. 14A) acquired bythe main image capturing system C1 includes the surface A3 of the objectA and the surface B3 of the object B. However, the captured image (FIG.14B) acquired by the sub image capturing system C2 does not includethese surfaces A3 and B3. In other words, the captured image acquired bythe sub image capturing system C2 does not include object areascorresponding to the surface A3 of the object A and the surface B3 ofthe object B that are included in the captured image acquired by themain image capturing system C1. Therefore, is impossible to calculatethe above-described displacement amounts of the object images. Namely, aproblem occurs that object distances of the object areas, in thecaptured image acquired by the main image capturing system C1,respectively including the surface A3 of the object A and the surface B3of the object B cannot be calculated.

A substantial problem does not occur if such an object area where theobject distance cannot be calculated is included in the captured imageacquired by the sub image capturing system C2. However, the object areawhere the object distance cannot be calculated in the captured imageacquired by the main image capturing system C1 causes problems such asimpossibility of performing AF control for that area using informationon the object distance.

Thus, in order to solve the problems, each embodiment appropriatelyarranges the multiple sub optical systems included in the multiple subimage capturing systems.

Embodiment 1

FIG. 1 illustrates an image capturing apparatus 1 that is a firstembodiment (Embodiment 1) of the present invention, which is viewed froman object side. A main optical system 100 as an imaging optical systemwhose magnification is variable has a variable focal length (35mm-equivalent focal length) from 24 mm to 300 mm. A main image sensor200 has a rectangular image capturing area corresponding to the mainoptical system 100 and captures (photoelectrically converts) an objectimage formed by the main optical system 100. The main optical system 100and the main image sensor 200 constitute a main image capturing system.

Four sub optical systems 110 a, 110 b, 120 a and 120 b are each a fixedfocal length imaging optical system. Four sub image sensors (notillustrated) capture (photoelectrically convert) object imagesrespectively formed by the four sub optical systems 110 a, 110 b, 120 aand 120 b. Each sub optical system and the sub image sensorcorresponding thereto constitute a sub image capturing system. That is,the mage capturing apparatus 1 of this embodiment has four sub imagecapturing systems. The four sub image capturing systems includes twofirst sub image capturing systems respectively constituted by the suboptical systems 110 a and 110 b and the sub image sensors correspondingthereto and two second sub image capturing systems respectivelyconstituted by the sub optical systems 120 a and 120 b and the sub imagesensors corresponding thereto.

The main optical system 100 and the four sub optical systems 110 a, 110b, 120 a and 120 b are arranged such that optical axes thereof areparallel to one another. The sub optical systems (each hereinafterreferred to as “a first sub optical system”) 110 a and 110 b are each afixed focal length optical system having a focal length equal to a 35mm-equivalent focal length of 24 mm (that is, having a view angle equalto that) of the main optical system 100 at its wide-angle end. On theother hand, the sub optical systems (each hereinafter referred to as “asecond sub optical system”) 120 a and 120 b are each a fixed focallength optical system having a focal length equal to a middle focallength of the main optical system 100 at its middle zoom positionbetween its wide-angle end and telephoto end. That is, each of thesecond sub optical systems 120 a and 120 b has a view angle narrowerthan that of the main optical system 100 at the wide-angle end and widerthan that of the main optical system 100 at the telephoto end. Forexample, each of the second sub optical systems 120 a and 120 b has a 35mm-equivalent focal length of 150 mm.

The first sub optical systems 110 a and 110 b are arranged on both sidesacross a sectional plane (straight line) 301 including the optical axis100 a of the main optical system 100 and in areas further outside thanthe main optical system 100 (that is, further outside than two lines303) in a horizontal direction orthogonal to the sectional plane 301.

The sectional plane 301 includes midpoints of two long sides of therectangular image capturing area of the main image sensor 200 and theoptical axis 100 a of the main optical system 100. Arranging the firstsub optical systems 110 a and 110 b as described above enables avoidinggeneration of the above-described occlusion area over an entirehorizontal area of a captured image acquired through the main opticalsystem 100 (that is, over an entire horizontal image capturing viewangle of the main image capturing system).

The second sub optical systems 120 a and 120 b are arranged on bothsides across a sectional plane (straight line) 302 including the opticalaxis 100 a of the main optical system 100 and in areas further outsidethan the main optical system 100 (that is, further outside than twolines 304) in a vertical direction orthogonal to the sectional plane302.

The sectional plane 302 includes midpoints of two short sides of therectangular image capturing area of the main image sensor 200 and theoptical axis 100 a of the main optical system 100. Arranging the secondsub optical systems 120 a and 120 b as described above enables avoidinggeneration of the above-described occlusion area over an entire verticalarea of the captured image acquired through the main optical system 100(that is, over an entire vertical image capturing view angle of the mainimage capturing system).

Avoiding the generation of the occlusion area enables successfullyperforming an image combination process and an object distancecalculation process, which are described later, in the image capturingview angle of the main image capturing system.

Next, description will be made of the principal of elimination of theocclusion area over the entire image capturing view angle of the mainimage capturing system by using captured images acquired by the main andsub image capturing systems. FIG. 3B illustrates a captured image(hereinafter referred to as “a main image”) acquired by image capture ofthe objects A and B, which are illustrated in FIG. 13, through the mainimage capturing system having the image capturing view angle at thewide-angle end. FIGS. 3A and 3C respectively illustrate two capturedimages (each hereinafter referred to as “a sub image”) acquired by imagecapture of the objects A and B through one first sub image capturingsystem (sub optical system 110 b) and the other first sub imagecapturing system (sub optical system 110 a). Each of these main and subimages includes, due to the difference of viewpoint positions asdescribed with referring to FIGS. 14A and 14B, a captured (included)part and a non-captured (non-included) part of an identical object.

However, surfaces A1 and B1 included in the main image as a base imageillustrated in FIG. 3B are included in both the two sub imagesillustrated in FIGS. 3A and 3C. On the other hand, surfaces A2 and B3included in the main image are included in the sub image illustrated inFIG. 3C, and the surfaces A3 and B3 are included in the sub imageillustrated in FIG. 3A. Namely, all surfaces of the objects A and Bincluded in the main image are included in at least one of the two subimages.

As described above, arranging the first sub optical systems 110 a and110 b on the both sides across the sectional plane 301 (and in the areasfurther horizontally outside than the main optical system 100) enableseliminating the occlusion area over the entire horizontal imagecapturing view angle of the main image capturing system. Similarly,arranging the second sub optical systems 120 a and 120 b on the bothsides across the sectional plane 302 (and in the areas furthervertically outside than the main optical system 100) enables eliminatingthe occlusion area over the entire vertical image capturing view angleof the main image capturing system.

FIG. 1 illustrates a most typical arrangement example of the mainoptical system 100 and the sub optical systems 110 a, 110 b, 120 a and120 b. The optical axes of the first sub optical systems 110 a and 110 band the optical axis 100 a of the main optical system 100 are located onthe identical straight line 302. Furthermore, the optical axes of thesecond sub optical systems 120 a and 120 b and the optical axis 100 a ofthe main optical system 100 are located on the identical straight line301. However, it is not necessarily needed that the main and sub opticalsystems be located on such an identical straight line. In other words,as long as the sub image optical systems are arranged on the both sidesacross the sectional plane including the optical axis 100 a of the mainoptical system 100 and thereby the occlusion area is eliminated over theentire vertical image capturing view angle of the main image capturingsystem, it is not needed that the main and sub optical systems belocated on an identical straight line.

FIG. 2 illustrates an electrical configuration of the image capturingapparatus 1 of this embodiment. The image capturing apparatus 1 includesan image capturing system constituted by the one main image capturingsystem and the four sub image capturing systems, an A/D converter 11, animage processor 12, an image combiner 13, an information inputter 14, animage capture controller 15, an image recording medium 16, a systemcontroller 17, a display device 18 and a distance information calculator19. The A/D converter 11 converts analog image capturing signals outputfrom the main image sensor 200 and the four sub image sensors (notillustrated) into digital image capturing signals to supply theconverted digital image signals to the image processor 12. The imageprocessor 12 performs on the digital image signals from the A/Dconverter 11 a pixel interpolation process, a color conversion process,a digital zoom process and others to produce captured images (one mainimage and four sub images). Furthermore, the image processor 12 performsvarious calculation processes using the captured images to sendcalculation results to the system controller 17. The image processor 12further includes the image combiner 13. The image combiner 13 sets themain image acquired by the main image capturing system, which serves asa base viewpoint, to a base image for image combination and combines atleast two of the four sub images acquired by the four sub imagecapturing systems with the base image. The image combination uses theblock matching method describe later.

The information inputter 14 detects information on an image capturingcondition (such as an aperture value and an exposure time) selected andinput by a user to supply data thereof to the system controller 17. Theimage capture controller 15 controls, depending on commands from thesystem controller 17, a movement of a focus lens included in eachoptical system, an aperture stop (aperture value) in each optical systemand a photoelectric conversion operation (exposure time) of the imagesensor of each image capturing system. The image recording medium 16records the captured images produced by the image processor 12 and afile header of an image file containing the captured images. The displaydevice 18 temporarily displays a preview image of the captured imageimmediately after image capture and displays the recorded capturedimages. The display device 18 further displays selectable menu items anda selectable focal length (view angle) range. The display device 18 isconstituted by a liquid crystal display device or the like.

The distance information calculator 19 includes a base image selector20, a corresponding point extractor 21 and a parallax amount calculator22. The base image selector 20 selects the main image acquired by themain image capturing system as a base image for object distanceinformation calculation. The corresponding point extractor 21 extractspixels (hereinafter referred to as “corresponding points”) correspondingto each other from paired ones of the main image and four sub images.The parallax amount calculator 22 calculates parallax amounts of all thecorresponding points extracted by the corresponding point extractor 21.The distance information calculator 19 calculates, using the calculatedparallax amounts, object distances in the entire main image (that is. inthe entire image capturing view angle of the main image capturingsystem).

Next, description will be made of an image capture/image combinationprocess performed by the system controller 17 and the image processor 12(image combiner 13) with referring to a flowchart of FIG. 4. The systemcontroller 17 and the image processor 12 each constituted by a computerexecute the image capture/image combination process according to animage capture/image combination program as a computer program. In thefollowing description, a step is abbreviated as “S”. The systemcontroller 17 serves as a selector, and the image processor 12 serves asa processor.

First, at S100, the system controller 17 acquires the information on theimage capturing condition and others input by the user through theinformation inputter 14. In response to input of an image captureinstruction from an image capture start switch (not illustrated)operated by the user, the system controller 17 transfers the informationon the image capturing condition and others to the image capturecontroller 15. The image capturing condition includes the imagecapturing view angle, the aperture value, the exposure time (shutterspeed) and others of the main image capturing system. The image capturecontroller 15 controls, depending on the input image capturingcondition, the image capturing view angle (zoom state) of the main imagecapturing system, the aperture values and the shutter speeds of the mainand sub image capturing systems and others.

Next, at S101, the system controller 17 selects, depending on the inputimage capturing condition, the first sub image capturing systems or thesecond sub image capturing systems as the sub image capturing systems tobe used for image capture. The selection of the sub image capturingsystems depends on the information on the image capturing view angle setfor the main image capturing system (hereinafter referred to as “a mainimage capturing view angle”). The system controller 17 performs imagecombination on the entire main image acquired by the main imagecapturing system and therefore selects the sub image capturing systemswhose image capturing view angle (hereinafter referred to as “a subimage capturing view angle”) is equal to or wider than that of the mainimage capturing view angle.

Next, at S102, the system controller 17 causes, through the imagecapture controller 15, the main image capturing system and the sub imagecapturing systems selected at S101 to perform image capture (exposure ofthe main and sub image sensors) and causes the image processor 12 toproduce captured images. Thereby, one main image and two sub images areacquired as parallax images mutually having parallaxes. In producingeach image, the image processor 12 may correct a brightness level and awhite balance of each image, which enables reducing brightnessunevenness and color unevenness of a combined image produced by theimage combination performed later.

Next, at S103, the system controller 17 determines whether or not themain image capturing view angle is equal to the sub image capturing viewangle of the selected sub image capturing systems. If the main imagecapturing view angle is equal to the sub image capturing view angle, thesystem controller 17 proceeds to S104, and otherwise proceeds to S105.

At S104, the system controller 17 causes the image processor 12 tocombine the sub images corresponding to the image capturing view angleequal to that of the main image with the main image as the base imagefor the image combination.

Next, description will be made of a method for combining the main andsub images performed by the image processor 12. Description herein willbe made of a case where the main image capturing view angle is set toits wide-angle end. In this embodiment, the image processor 12 uses themain image (FIG. 3B) as the base image and extracts, from the sub images(FIGS. 3A and 3C) as reference images acquired by the first sub imagecapturing systems (first sub optical systems 110 a and 110 b), objectareas corresponding to an object area included in the base image tocombine the extracted object areas with the base images.

First, description will be made of a method for extracting, from the subimage (reference image), the object area (corresponding point)corresponding to the object area included in the main image (baseimage), with reference to FIGS. 5A and 5B.

FIG. 5A illustrates the base image 501 that is the main imageillustrated in FIG. 3B, and FIG. 5B illustrates the reference image 502that is the sub image illustrated in FIG. 3A acquired through the suboptical system 110 b. The following description uses image coordinates(X,Y) indicating horizontal and vertical positions (pixel) in eachimage. The image coordinates (X,Y) are based on an origin located at anupper left position in the images illustrated in FIGS. 5A and 5B. Inaddition, in the following description, F1(X,Y) represents a brightnessvalue of a pixel (X,Y) in the base image 501, and F2(X,Y) represents abrightness value of a pixel (X,Y) in the reference image 502.

A pixel (hatched) in the reference image 502 corresponding to anarbitrary pixel (hatched) (X,Y) in the base image 501 is determined bysearching in the reference image 502 for a brightness value most similarto F1(X,Y) in the base image 501. However, it is difficult to simplysearch for a pixel (corresponding point; hereinafter referred to as “acorresponding pixel”) whose brightness value is most similar to that ofthe arbitrary pixel, so that a block matching method using pixels nearthe pixel (X,Y) is employed to search for the corresponding pixel.

For example, description will be made of the block matching method in acase where a block size is 3.

In the base image 501, three pixels including an arbitrary pixel (X,Y)and its left and right pixels (X−1,Y) and (X+1,Y) respectively havebrightness values F1(X,Y), F1(X−1,Y) and F1(X+1,Y). On the other hand,in the reference image 502, three pixels shifted by k from thecoordinates (X,Y), (X−1,Y) and (X+1,Y) respectively have brightnessvalues F2(X+k,Y), F2(X+k−1,Y) and F2(X+k+1,Y). A similarity E to thepixel (X,Y) in the base image 501 is defined by following expression(4).

$\begin{matrix}{E = {{\lbrack {{F\; 1( {X,Y} )} - {F\; 2( {{X + k},Y} )}} \rbrack + \lbrack {{F\; 1( {{X - 1},Y} )} - {F\; 2( {{X + k - 1},Y} )}} \rbrack + \lbrack {{F\; 1( {{X + 1},Y} )} - {F\; 2( {{X + k + 1},Y} )}} \rbrack} = {\sum\limits_{j = {- 1}}^{1}\;\lbrack {{F\; 1( {{X + j},Y} )} - {F\; 2( {{X + k + j},Y} )}} \rbrack}}} & (4)\end{matrix}$

Calculating the similarities E while sequentially changing the value ofk in above expression (4) provides a pixel (X+k,Y) whose similarity E issmallest. This pixel (X+k,Y) is the corresponding pixel in the referenceimage 502 that corresponds to the pixel (X,Y) in the base image 501.Although the above description was made of the method for thecorresponding point extraction between the images having a parallax inthe horizontal direction, a similar method can be used for correspondingpoint extraction between images having a parallax in the verticaldirection or an oblique direction.

Combining the object area as the corresponding point thus acquired withthe base image pixel by pixel enables reducing a noise level in the baseimage and improving an image quality of an output combined image. Thesub image illustrated in FIG. 3C acquired through the sub optical system110 a does not include object areas corresponding to the surfaces A3 andB3 in the main image illustrated in FIG. 3B, so that these surfaces A3and B3 are not combined from the sub image acquired through the suboptical system 110 a with the base image. Furthermore, the sub imageillustrated in FIG. 3A acquired through the sub optical system 110 bdoes not include object areas corresponding to the surfaces A2 and B2 inthe main image illustrated in FIG. 3B, so that these surfaces A2 and B2are not combined from the sub image acquired through the sub opticalsystem 110 b with the base image. Such occlusion areas cannot becombined from the sub images. However, most of other areas than theocclusion areas can be combined from the sub images acquired by imagecapture from different viewpoint positions with the main image, whichenables reducing the noise level of the combined image as a whole.

In a case where the parallax between the base and reference images istoo large and shapes thereof are too different from each other to usethe block matching method, an affine transform or the like may beperformed on the reference image, and thereafter the image combinationmay be performed using the block matching method.

A similar method to the above-described image combination method cancombine the sub image acquired by the second sub image capturing systems(sub optical systems 120 a and 120 b) with the main image.

The system controller 17 proceeding from S103 to S105, because the subimage capturing view angle is different from the main image capturingview angle, performs a trimming/enlargement process on a partial imagearea in the sub image (reference image) such that the enlarged partialimage area becomes an image corresponding to the main image capturingview angle. In this description, the main image capturing view anglecorresponds to a 35 mm-equivalent focal length of 50 mm.

FIGS. 6A and 6B illustrate sub images acquired by image capture of theobjects A and B illustrated in FIG. 13 through the first sub imagecapturing systems (first sub optical systems 110 b and 110 a).Furthermore, FIG. 6C illustrates a main image acquired by image captureof the objects A and B through the main image capturing system (mainoptical system 100). Moreover, FIG. 6D illustrates an image (hereinafterreferred to as “an enlarged image”) acquired by trimming and enlarging apartial image area including the objects A and B in the sub imageillustrated in FIG. 6B such that the enlarged image corresponds to themain image capturing view angle. Although not illustrated, a partialimage area including the objects A and B in the sub image illustrated inFIG. 6A is also trimmed and enlarged such that the enlarged partialimage area (enlarged image) corresponds to the main image capturing viewangle.

Next, at S106, the system controller 17 causes the image processor 12 tocombine the enlarged image acquired through the trimming/enlargementprocess at S105 with the main image (base image).

Description will be made of a method for combining the reference imagecorresponding to a different image capturing view angle from that of thebase image with the base image. The reference images (sub images)illustrated in FIGS. 6A and 6B acquired with a wider image capturingview angle than that for acquiring the base image (main image)illustrated in FIG. 6C include the objects A and B whose sizes aredifferent from those in the base image. Therefore, the image processor12 at S105 trims and enlarges the area including the objects A and B inthe sub image area of FIG. 6B such that the enlarged area becomes animage corresponding to an image capturing view angle equal to the mainimage capturing view angle to produce the enlarged image illustrated inFIG. 6D. Such a trimming/enlargement process deteriorates a resolutionof the enlarged image as compared with that of the original sub image,but enables making the sizes of the objects A and B in the enlargedimage equal to those in the main image. Thereafter, the image processor12 combines, as at S104, the enlarged image as a new reference imagewith the base image to produce a combined image.

In this embodiment, as described above, the first sub optical systems110 a and 110 b are arranged on the both sides across the sectionalplane 301. With this arrangement, as illustrated in FIGS. 6A to 6C, allthe object areas included in the main image acquired by the main imagecapturing system are included in the two sub images acquired by the twofirst sub image capturing systems including the first sub opticalsystems 110 a and 110 b. That is, an occlusion area that is included inthe main image as the base image but is not included in either of thesub images is not generated. Accordingly, all the object areas includedin the base image can be combined from at least one of the sub imagesacquired by the two first sub image capturing systems.

A similar effect thereto can be acquired by the arrangement of thesecond sub optical systems 120 a and 120 b on the both sides across thesectional plane 302. That is, all the object areas included in the mainimage acquired by the main image capturing system are included in thetwo sub images acquired by the two second sub image capturing systemsincluding the second sub optical systems 120 a and 120 b. Accordingly,all the object areas included in the base image can be combined from atleast one of the sub images acquired by the two second sub imagecapturing systems.

After the combined image is thus produced at S104 or S106, the systemcontroller 17 at S107 stores the combined image to the image recordingmedium 16 and then ends this process.

Next, description will be made of an image capture/object distancecalculation process performed by the system controller 17 and thedistance calculator 19 with referring to a flowchart of FIG. 7. First,description will be made of a process in a case of using the parallaximages acquired by the main and first sub image capturing systems.

The system controller 17 and the object distance calculator 19 eachconstituted by a computer execute the image capture/object distancecalculation process according to an image capture/object distancecalculation program as a computer program. In the following description,a step is abbreviated as “S”.

Processes at S200 to S202 are the same as those at S100 to S102illustrated in FIG. 4, which relate to the image capture and theproduction of the parallax images, so that description thereof isomitted. In addition, processes at S203 and S204 are the same as thoseat S103 and S105 illustrated in FIG. 4, which relate to thetrimming/enlargement process, so that description thereof is omitted.The system controller 17 proceeding from S203 or S204 to S205 causes thebase image selector 20 to select the main image as a base image forobject distance calculation. Furthermore, the system controller 17causes the corresponding point extractor 21 to extract correspondingpoints between the base image and the reference images as the sub imagesor the enlarged images. The corresponding points are, when, for example,the base image and the reference image that are two parallax imagesinclude the object A illustrated in FIG. 13, pixels constituting thesame point of the object A in the two parallax images. The correspondingpoint extraction method described in the image combination process isused in this process without change, so that description thereof isomitted.

Next, at S206, the system controller 17 causes the parallax amountcalculator 22 to calculate parallax amounts at the respectivecorresponding points extracted at S205. The parallax amount iscalculated as a difference between a position of each pixel (base pixel)in the base image and that of a pixel (corresponding pixel) in thereference image corresponding to the base pixel; the positions of thebase pixel and the corresponding pixel were acquired by theabove-described block matching method.

Next, at S207, the system controller 17 causes the distance informationcalculator 19 to calculate an object distance of each object included inthe main image. The distance information calculator 19 calculates anobject distance to each object using expression (3) using the parallaxamount calculated at S206, the known focal length of the main opticalsystem 100 and the base length between the sub optical systems 110 a and110 b.

Next, at S208, the system controller 17 records information (objectdistance information) on the object distance calculated at S207,together with data of the parallax images acquired at S202, to the imagerecording medium 16 and then ends this process.

Although the above description was made of the calculation of the objectdistance in the case of using the first sub optical systems 110 a and110 b, a similar process using the second sub optical systems 120 a and120 b to the above-described process also can calculate an objectdistance to each object.

Description will be made of advantages of the object distancecalculation using the second sub optical systems 120 a and 120 b havingthe middle focus length (middle view angle) between the wide-angle endand telephoto end of the main optical system 100. In a case where atS200 in FIG. 7 the focal length of the main optical system 100 is set to150 mm through the information inputter 14, at S201 the second sub imagecapturing system having the second sub optical systems 120 a and 120 bwhose view angles (focal lengths) are equal to that of the main opticalsystem 100 are selected. In this case, at S201 the first sub imagecapturing systems whose image capturing view angles are wider than thatof the main image capturing system may be selected. However, since theselection of the first sub image capturing systems has a disadvantagedescribed later, it is desirable to select the second sub opticalsystems 120 a and 120 b whose view angles are nearest, that is, equal to(or wider than) that of the main optical system 100.

FIGS. 8A to 8C illustrate two sub images and a main image respectivelyacquired at S202 by image capture of the objects A and B illustrated inFIG. 13 through the two second sub image capturing systems (second suboptical systems 120 a and 120 b) and the main image capturing system(main optical system 100). FIG. 8D illustrates, if the two first subimage capturing systems are selected, a sub image acquired by imagecapture of the objects A and B through one of the first sub imagecapturing systems (the first sub optical system 110 a).

As illustrated in FIGS. 8A to 8D, since the image capturing view anglesof the second sub image capturing systems are equal to that of the mainimage capturing system, the system controller 17 proceeds from S203 inFIG. 7 to the corresponding point extraction process (S205). Asdescribed above, the two second sub optical systems 120 a and 120 b arearranged on the both sides across the sectional plane 302. Therefore, asillustrated in FIGS. 8A to 8C, the objects A and B included in the mainimage are both included in the two sub images, that is, no occlusionarea is generated. Furthermore, since it is not necessary to trim andenlarge the areas of the objects A and B in the sub images acquired bythe second sub image capturing systems, no resolution deteriorationoccurs and a good corresponding point extraction is performed, whichprovides a good calculation result of the parallax amounts at S206.

However, if the first sub image capturing systems are selected, asillustrated in FIG. 8D, the sizes of the objects A and B in each subimage acquired by the first sub image capturing system is significantlydifferent from those in the main image, which makes it necessary toenlarge the area of the objects A and B in the sub image six times.Therefore, the resolution of the enlarged image is significantlydeteriorated, which may cause large errors in the corresponding pointextraction and the parallax amount calculation at S205 and S206. As aresult, the object distance may be erroneously calculated and therebyaccuracy of the AF may be deteriorated. That is, using the sub imagecapturing systems whose image capturing view angles are equal to orwider than the middle image capturing view angle between the wide-angleend and telephoto end of the main image capturing system enablesaccurately calculating the object distance over the entire imagecapturing view angle of the main image capturing system.

As described above, in this embodiment, the sub optical systems (110 aand 110 b or 120 a and 120 b) each constituted by a fixed focal lengthoptical system are arranged on the both sides across the sectional plane(301 or 302) including the optical axis 100 a of the main optical system100. This arrangement enables avoiding the generation of the occlusionarea over the entire image capturing view angle of the main imagecapturing system and thereby achieves an image capturing apparatuscapable of performing a good image combination process and a good objectdistance calculation process.

Furthermore, the sub optical systems have the middle focal length (viewangle) between the focal lengths at the wide-angle end and telephoto endof the main optical system, which enables accurately calculating theobject distance over the entire image capturing view angle of the mainimage capturing system.

Moreover, using the configuration of this embodiment can realize variousimage capturing modes. For example, in a high dynamic range mode,performing image capture by the multiple sub image capturing systemsunder mutually different exposure conditions and combining multiple subimages acquired thereby with a main image acquired by the main imagecapturing system enables providing a wide dynamic range combined image.In a blur adding mode, adding blur to a background depending on theobject distance calculated as described above enables producing an imagein which a main object is highlighted. In a background removing mode,using the object distance calculated as described above enablesproducing an image in which a background other than a main object isremoved. In a stereo image capturing mode, acquiring right and leftparallax images by the main and sub image capturing systems horizontallyarranged enables producing a stereoscopically viewable image by usingone of the parallax images corresponding to a narrow view angle and partof the other one thereof corresponding to a wide view angle.

Although this embodiment described the case where the sub imagecapturing systems are each a fixed focal length optical system, the subimage capturing systems may be each a magnification variable (variablefocal length) optical system.

Embodiment 2

Next, description will be made of a second embodiment (Embodiment 2) ofthe present invention. FIG. 9 illustrates an image capturing apparatus 2of this embodiment, which is viewed from an object side.

A main optical system 100 is, as that in Embodiment 1, a magnificationvariable imaging optical system having a variable focal length (35mm-equivalent focal length) from 24 mm to 300 mm. A main image sensor200 has a rectangular image capturing area corresponding to the mainoptical system 100 and captures an object image formed by the mainoptical system 100. The main optical system 100 and the main imagesensor 200 constitute a main image capturing system.

Four sub optical systems 130 a, 130 b, 140 a and 140 b are each a fixedfocal length imaging optical system. Four sub image sensors (notillustrated) capture object images respectively formed by the four suboptical systems 130 a, 130 b, 140 a and 140 b. Each sub optical systemand the sub image sensor corresponding thereto constitute a sub imagecapturing system. That is, the mage capturing apparatus 2 of thisembodiment has four sub image capturing systems. The four sub imagecapturing systems includes two first sub image capturing systemsrespectively constituted by the sub optical systems 130 a and 130 b andthe sub image sensors corresponding thereto and two second sub imagecapturing systems respectively constituted by the sub optical systems140 a and 140 b and the sub image sensors corresponding thereto. Themain optical system 100 and the four sub optical systems 130 a, 130 b,140 a and 140 b are arranged such that optical axes thereof are parallelto one another. The sub optical systems (each hereinafter referred to as“a first sub optical system”) 130 a and 130 b are each a fixed focallength optical system having a focal length equal to a 35 mm-equivalentfocal length of 24 mm (that is, having a view angle equal to that) ofthe main optical system 100 at its wide-angle end. On the other hand,the sub optical systems (each hereinafter referred to as “a second suboptical system”) 140 a and 140 b are each a fixed focal length opticalsystem having a focal length equal to a middle focal length of the mainoptical system 100 at its middle zoom position between its wide-angleend and telephoto end. That is, each of the second sub optical systems140 a and 140 b has a view angle narrower than that of the main opticalsystem 100 at the wide-angle end and wider than that of the main opticalsystem 100 at the telephoto end. For example, each of the second suboptical systems 140 a and 140 b has a 35 mm-equivalent focal length of100 mm. The first sub optical systems 130 a and 130 b are arranged onboth sides across a sectional plane (straight line) 301 including theoptical axis 100 a of the main optical system 100 and in areas furtheroutside than the main optical system 100 (that is, further outside thantwo lines 303) in a horizontal direction orthogonal to the sectionalplane 301. The sectional plane 301 includes midpoints of two long sidesof the rectangular image capturing area of the main image sensor 200 andthe optical axis 100 a of the main optical system 100. Arranging thefirst sub optical systems 130 a and 130 b as described above enablesavoiding generation of the above-described occlusion area over an entirehorizontal area of a captured image acquired through the main opticalsystem 100 (that is, over an entire horizontal image capturing viewangle of the main image capturing system).

The second sub optical systems 140 a and 140 b are arranged on bothsides across a sectional plane (straight line) 302 including the opticalaxis 100 a of the main optical system 100 and in areas further outsidethan the main optical system 100 (that is, further outside than twolines 304) in a vertical direction orthogonal to the sectional plane302.

The sectional plane 302 includes midpoints of two short sides of therectangular image capturing area of the main image sensor 200 and theoptical axis 100 a of the main optical system 100. Arranging the secondsub optical systems 140 a and 140 b as described above enables avoidinggeneration of the above-described occlusion area over an entire verticalarea of the captured image acquired through the main optical system 100(that is, over an entire vertical image capturing view angle of the mainimage capturing system).

Also in this embodiment, avoiding the generation of the occlusion areaenables successfully performing an image combination process and anobject distance calculation process over the entire horizontal andvertical image capturing view angles of the main image capturing system.In FIG. 9, the optical axes of the first sub optical systems 130 a and130 b and the optical axis 100 a of the main optical system 100 arelocated on the identical straight line 302. Furthermore, the opticalaxes of the second sub optical systems 140 a and 140 b and the opticalaxis 100 a of the main optical system 100 are located on the identicalstraight line 301.

The above-described configuration is identical to that of Embodiment 1.However, in this embodiment, an inter-optical axis distance (baselength) between the optical axis of the first sub optical system 130 aand the optical axis 100 a of the main optical system 100 is differentfrom that between the optical axis of the first sub optical system 130 band the optical axis 100 a of the main optical system 100. Similarly, aninter-optical axis distance (base length) between the optical axis ofthe second sub optical system 140 a and the optical axis 100 a of themain optical system 100 is different from that between the optical axisof the second sub optical system 140 b and the optical axis 100 a of themain optical system 100.

More specifically, the first sub optical system 130 a and the second suboptical system 140 a are arranged near the main optical system 100, andon the other hand, the first sub optical system 130 b and the second suboptical system 140 b are disposed away from the main optical system 100.

Description will be made of an effect of such an arrangement of the suboptical systems with the different inter-optical axis distances withreferring to FIGS. 10A to 10C. FIG. 10B illustrates a main imageacquired by image capture of the objects A and B illustrated in FIG. 13through the main image capturing system in which the view angle of themain optical system 100 is set to one at the wide-angle end. FIGS. 10Aand 10C illustrate sub images acquired by image capture of the objects Aand B illustrated in FIG. 13 through the sub image capturing systemincluding the first sub optical system 130 b and the sub image capturingsystem including the first sub optical system 130 a.

When object distances that are distances to the objects A and B areshort (near), a long base length between the main optical system 100 andthe first sub optical system 130 b excessively increases a parallaxamount as illustrated in FIG. 10A, which causes the object A to beprotruded outside the image capturing view angle. If the entire object Ais not included in the sub image (reference image) acquired by the subimage capturing system, the above-mentioned corresponding pointextraction process cannot be performed, thereby making it impossible tocalculate the object distance.

However, in this embodiment, providing the first sub optical system 130a near the main optical system 100 and thereby shortening the baselength therebetween enables reducing the parallax amount even when theobject distance is short. Therefore, this embodiment enables, asillustrated in FIG. 10C, causing the sub image acquired by the first subimage capturing system including the first sub optical system 130 a toinclude the entire objects A and B. Accordingly, when capturing a closedistance object, using the main image acquired by the main imagecapturing system and the sub image acquired by the first sub imagecapturing system including the first sub optical system 130 a enablescalculating object distances over the entire image capturing view angleof the main image capturing system.

On the other hand, when the object distances to the objects A and B arelong (far), a short base length between the main optical system 100 andthe first sub optical system 130 a excessively decreases the parallaxamount, which deteriorates accuracy of the object distance calculation.

However, in this embodiment, providing the first sub optical system 130b away from the main optical system 100 and thereby lengthening the baselength therebetween enables increasing the parallax amount even when theobject distance is long. Accordingly, when capturing a far-distanceobject, using the main image acquired by the main image capturing systemand the sub image acquired by the first sub image capturing systemincluding the first sub optical system 130 b enables accuratelycalculating object distances.

As described above, in this embodiment, the multiple sub optical systems130 a, 130 b, 140 a and 140 b are arranged such that the inter-opticalaxis distance between at least one of the sub optical systems 130 a, 130b, 140 a and 140 b and the main optical system 100 is different fromthat between at least another one thereof and the main optical system100. This arrangement enables providing accurate object distanceinformation in a wide object distance range.

Embodiment 3

Next, description will be made of a third embodiment (Embodiment 3) ofthe present invention. FIG. 11 illustrates an image capturing apparatus3 of this embodiment, which is viewed from an object side. A mainoptical system 100 is, as those in Embodiments 1 and 2, a magnificationvariable imaging optical system having a variable focal length (35mm-equivalent focal length) from 24 mm to 300 mm. A main image sensor200 has a rectangular image capturing area corresponding to the mainoptical system 100 and captures an object image formed by the mainoptical system 100. The main optical system 100 and the main imagesensor 200 constitute a main image capturing system.

Two sub optical systems 150 a and 150 b are each a fixed focal lengthimaging optical system.

Two sub image sensors (not illustrated) capture object imagesrespectively formed by the two sub optical systems 150 a and 150 b. Eachsub optical system and the sub image sensor corresponding theretoconstitute a sub image capturing system. That is, the image capturingapparatus 3 of this embodiment has two sub image capturing systems. Themain optical system 100 and the two sub optical systems 150 a and 150 bare arranged such that optical axes thereof are parallel to one another.The sub optical system 150 a is a fixed focal length optical systemhaving a 35 mm-equivalent focal length of 20 mm, which is shorter (thatis, having a view angle wider) than a 35 mm-equivalent focal length of24 mm of the main optical system 100 at its wide-angle end. On the otherhand, the sub optical system 150 b is a fixed focal length opticalsystem having a focal length equal to a focal length of the main opticalsystem 100 at a middle zoom position between its wide-angle end andtelephoto end. That is, the sub optical system 150 b has a view anglenarrower than that of the main optical system 100 at the wide-angle endand wider than that of the main optical system 100 at the telephoto end.For example, the sub optical system 150 b has a 35 mm-equivalent focallength of 100 mm.

The sub optical systems 150 a and 150 b are arranged on both sidesacross a sectional plane (straight line) 301 including the optical axis100 a of the main optical system 100 and in areas further outside thanthe main optical system 100 (that is, further outside than two lines303) in a horizontal direction orthogonal to the sectional plane 301.The sectional plane 301 includes midpoints of two long sides of therectangular image capturing area of the main image sensor 200 and theoptical axis 100 a of the main optical system 100. Arranging the suboptical systems 150 a and 150 b as described above enables avoidinggeneration of the above-described occlusion area over an entirehorizontal area of a captured image acquired through the main opticalsystem 100 (that is, over an entire horizontal image capturing viewangle of the main image capturing system).

In FIG. 11, the optical axes of the sub optical systems 150 a and 150 band the optical axis 100 a of the main optical system 100 are located onan identical straight line 302.

This embodiment is different from Embodiments 1 and 2 in that part ofthe sub optical systems has a wider image capturing view angle than thatof the main image capturing system at its wide-angle end. Thus, thisembodiment enables a user to observe a wider object range through thesub image capturing system than that through the main image capturingsystem, which enables utilizing the sub image capturing system forfollowing a moving object and for setting an appropriate image capturingview angle of the main image capturing system.

Furthermore, in this embodiment, the two sub image capturing systemshave mutually different focal lengths (image capturing view angles).Therefore, only when the view angle of the main optical system 100 isset narrower than that (focal length 100 mm) of the sub optical system150 b, the generation of the occlusion area can be avoided over theentire image capturing view angle of the main image capturing system.Specifically, a partial image area of a sub image acquired by the subimage capturing system including the sub optical system 150 a is trimmedand enlarged such that a size of an object becomes equal to thatincluded in a sub image acquired by the sub image capturing systemincluding the sub optical system 150 b.

Thereafter, subsequent processes can be performed as in Embodiment 1. Inthis embodiment, as in Embodiment 1, having the sub optical system 150 bwhose focal length is equal to the middle focal length of the mainoptical system 100 between its wide-angle end and telephoto end enables,irrespective of a reduced total number of the sub image capturingsystems, accurately calculating object distances by using a telephotoside image capturing view angle of the main image capturing system.

Embodiment 4

Next, description will be made of a fourth embodiment (Embodiment 4) ofthe present invention. FIG. 12 illustrates an image capturing apparatus4 of this embodiment, which is viewed from an object side. A mainoptical system 100 is, as those in Embodiments 1 to 3, a magnificationvariable imaging optical system having a variable focal length (35mm-equivalent focal length) from 24 mm to 300 mm. A main image sensor200 has a rectangular image capturing area corresponding to the mainoptical system 100 and captures an object image formed by the mainoptical system 100. The main optical system 100 and the main imagesensor 200 constitute a main image capturing system.

Six sub optical systems 160 a, 160 b, 170 a, 170 b, 180 a and 180 b areeach a fixed focal length imaging optical system.

Six sub image sensors (not illustrated) capture object imagesrespectively formed by the six sub optical systems 160 a, 160 b, 170 a,170 b, 180 a and 180 b. Each sub optical system and the sub image sensorcorresponding thereto constitute a sub image capturing system. That is,the mage capturing apparatus 4 of this embodiment has six sub imagecapturing systems. The six sub image capturing systems includes twofirst sub image capturing systems respectively constituted by the suboptical systems 160 a and 160 b and the sub image sensors correspondingthereto and four second sub image capturing systems respectivelyconstituted by the sub optical systems 170 a, 170 b, 180 a and 180 b andthe sub image sensors corresponding thereto.

In the following description, the two sub image capturing systemsincluding the sub optical systems 170 a and 170 b is referred to as“second sub image capturing systems”, and the two sub image capturingsystems including the sub optical systems 180 a and 180 b is referred toas “third sub image capturing systems”.

The main optical system 100 and the six sub optical systems 160 a, 160b, 170 a, 170 b, 180 a and 180 b are arranged such that optical axesthereof are parallel to one another. The sub optical systems (eachhereinafter referred to as “a first sub optical system”) 160 a and 160 bare each a fixed focal length optical system having a focal length equalto a 35 mm-equivalent focal length of 24 mm (that is, having a viewangle equal to that) of the main optical system 100 at its wide-angleend. On the other hand, the sub optical systems (each hereinafterreferred to as “a second sub optical system”) 170 a and 170 b are each afixed focal length optical system having a focal length equal to amiddle focal length of the main optical system 100 at its middle zoomposition between its wide-angle end and telephoto end. That is, each ofthe second sub optical systems 170 a and 170 b has a view angle narrowerthan that of the main optical system 100 at the wide-angle end and widerthan that of the main optical system 100 at the telephoto end. Forexample, each of the second sub optical systems 170 a and 170 b has a 35mm-equivalent focal length of 100 mm. Furthermore, the sub opticalsystems (each hereinafter referred to as “a third optical system”) 180 aand 180 b are each a fixed focal length optical system having a focallength equal to another middle focal length of the main optical system100 at its another middle zoom position between the wide-angle end andtelephoto end. That is, each of the second sub optical systems 170 a and170 b has a view angle narrower than that of the main optical system 100at the wide-angle end and wider than that of the main optical system 100at the telephoto end. For example, each of the second sub opticalsystems 180 a and 180 b has a 35 mm-equivalent focal length of 200 mm.The first sub optical systems 160 a and 160 b are arranged on both sidesacross a sectional plane (straight line) 301 including the optical axis100 a of the main optical system 100 and in areas further outside thanthe main optical system 100 (that is, further outside than two lines303) in a horizontal direction orthogonal to the sectional plane 301.The sectional plane 301 includes midpoints of two long sides of therectangular image capturing area of the main image sensor 200 and theoptical axis 100 a of the main optical system 100. Arranging the firstsub optical systems 160 a and 160 b as described above enables avoidinggeneration of the above-described occlusion area over an entirehorizontal area of a captured image acquired through the main opticalsystem 100 (that is, over an entire horizontal image capturing viewangle of the main image capturing system).

The second and third sub optical systems 170 a, 170 b, 180 a and 180 bare arranged on both sides across a sectional plane (straight line) 302including the optical axis 100 a of the main optical system 100 and inareas further outside than the main optical system 100 (that is, furtheroutside than two lines 304) in a vertical direction orthogonal to thesectional plane 302. The sectional plane 302 includes midpoints of twoshort sides of the rectangular image capturing area of the main imagesensor 200 and the optical axis 100 a of the main optical system 100.Arranging the second and third optical systems 170 a, 170 b, 180 a and180 b as described above enables avoiding generation of theabove-described occlusion area over an entire vertical area of thecaptured image acquired through the main optical system 100 (that is,over an entire vertical image capturing view angle of the main imagecapturing system).

In FIG. 12, the optical axes of the first sub optical systems 160 a and160 b and the optical axis 100 a of the main optical system 100 arelocated on the identical straight line 302. Furthermore, the opticalaxes of the third sub optical systems 180 a and 180 b and the opticalaxis 100 a of the main optical system 100 are located on the identicalstraight line 301.

On the other hand, the optical axes of the second sub optical systems170 a and 170 b are not located on the straight lines 301 and 302, butare located on a straight line 305 tilted with respect to the straightlines 301 and 302.

This embodiment has the sub image capturing systems having two imagecapturing view angles between the wide-angle end and telephoto end ofthe main image capturing system (the sub image capturing systems arehereinafter referred to as “two types of sub image capturing systems”).Having such two types of image capturing systems enables reducing anenlargement rate in trimming and enlarging a partial image area of a subimage acquired by any of the two types of sub image capturing systems.Therefore, this embodiment enables reducing resolution deterioration andthereby enables providing more accurate object distance information.

Each of the above-described embodiments enables avoiding disadvantagesdue to the generation of the occlusion area over the entire imagecapturing view angle of the main image capturing system.

In other words, each embodiment achieves an image capturing apparatuscapable of acquiring accurate object distance information over theentire image capturing view angle of the main image capturing system andof successfully combining multiple images acquired by the main andmultiple sub image capturing systems with one another.

Although the above embodiments described the case where the main andmultiple image capturing systems include separate image sensors (mainand multiple sub image sensors), a single image sensor may be used thathas a main image capturing area provided for the main image capturingsystem and multiple image capturing areas provided for the multiple subimage capturing systems.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-147640, filed on Jul. 27, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image capturing apparatus comprising: a mainimage capturing system configured to capture an object image formed by amain optical system, the main optical system being a zoom lens; andmultiple sub image capturing systems configured to respectively capturemultiple object images respectively formed by multiple sub opticalsystems, wherein the multiple sub optical systems are arranged on bothsides across a sectional plane including an optical axis of the mainoptical system, wherein at least one sub optical system included in themultiple sub optical systems is a fixed focal length optical systemwhose view angle is equal to or wider than that of the main opticalsystem at its wide-angle end, wherein at least one other sub opticalsystem included in the multiple sub optical systems is a fixed focallength optical system whose view angle is smaller than that of the mainoptical system at its wide-angle end and is equal to or wider than thatof the main optical system at its telephoto end, and wherein theapparatus further comprises a processor configured to acquire, using amain image produced by the main image capturing system and multiple subimages produced by the multiple sub image capturing systems, objectdistance information in an image capturing view angle of the main imagecapturing system.
 2. The image capturing apparatus according to claim 1,wherein the multiple sub optical systems are arranged on the both sidesacross the sectional plane and in areas further outside than the mainoptical system in a direction orthogonal to the sectional plane.
 3. Theimage capturing apparatus according to claim 1, wherein each of the suboptical systems is a fixed focal length optical system.
 4. The imagecapturing apparatus according to claim 1, wherein the multiple suboptical systems include: at least two first sub optical systems eachbeing a fixed focal length optical system whose view angle is equal toor wider than that of the main optical system at its wide-angle end; andat least two second sub optical systems each being a fixed focal lengthoptical system whose view angle is smaller than that of the main opticalsystem at its wide-angle end and is equal to or wider than that of themain optical system at its telephoto end.
 5. The image capturingapparatus according to claim 1, wherein an inter-optical axis distancebetween at least one of the multiple sub optical systems and the mainoptical system is different from that between at least another one ofthe sub optical systems and the main optical system.
 6. The imagecapturing apparatus according to claim 1, wherein optical axes of atleast two of the multiple sub optical systems and the optical axis ofthe main optical system are located on an identical straight line. 7.The image capturing apparatus according to claim 1, further comprising aselector configured to select from the multiple sub image capturingsystems, depending on a view angle of the main optical system, at leastone sub image capturing system used for acquiring the object distanceinformation.
 8. The image capturing apparatus according to claim 7,wherein the processor is configured to, when the image capturing viewangle of the main image capturing system is different from that of theselected sub image capturing system in acquiring the object distanceinformation, acquire the object distance information using an enlargedimage obtained by trimming and enlarging an image area corresponding tothe image capturing view angle from the sub image produced by theselected sub image capturing system.